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y  This  research  investigated  the  ability  of  computer  users,  dot 
programmers  and  non-programmers,  to  specify  f  r  obi  cm  solution.,  m  in¬ 
form  of  example-solutions.  This  ability  was  evaluated  a.>  a  function  or 
t  rj  complexity  of  the  processor,  i.e.  the  degree  of  generalization  of 
the  user  inputs,  the  complexity  of  the  problem,  arc  tne  complexity  of  a 
feedback-aids.  The  experimental  task  employed  in  this  study  required 
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A ,\AC  . 

Tnis  . nve.  ■.■u  ;  m  acbity  of  CGmpm;e-~  u.-ers 

both  programmers  and  nor-  ,rc.  ;m.mm._  m ,  to  specify  pro.;  ! cm 
solutions  m  too  form  of  ^xarr.p'^-i.r.i^c.or^.  Trus  ability  v/as  evalu 
as  a  fo notion  of  tnc  complexity  •  f  tn_  pr .'■censor ,  i.e,  me  ^e'.rec 
of  generalisation  of  tne  i npua.-,  fm.  complexity  of  the  groolem 

the  complexity  of  the  foe^oacK-aica> .  Tne  experimental  tea  x  em¬ 
ployed  in  tnis  study  requires  tne  snecif. cation  of  a  logic  for  tne 
formation  of  a  naval  task-force.  T  ~,e  performance  octr  of  pro-  . 
grammers  and  non-programme^-  decreased  witn  increasing  levels 
of  problem-complexity  ano  wit.n  reduced  processor-support .  For 
both  tne  groups,  erro^s-of-commission  were  relatively  infrequent 
compared  to  errors-of-ormssion .  it  was  found  that  tnc  degree  of 
processor-complexity  was  muen  more  influential  than  problem- 
complexity  in  predicting  performance  scores.  When  little  com¬ 
puter  generalization  of  usei — inp.n  was  provided,  performance  was 
significantly  lower  tnan  during  all  of  nor  experimental  conditions. 
Results  also  showed  boat  partic  i  pant-strategy  m  the  generation  of 

problem  solutions  was  a  significant  f cA o  v  o *r  i  n  c ir  I  o r' O"! ci in ^  o ,  ^ n o 

years-of-experience  and  years-of-education  were  not  found  to  be 
good  predictors  of  performance.  Tne  fceaoack-aids  were  shown 
to  be  most  effective  when  they  includes  tne  logic  implied  by  tne 
example-solutions.  These  experiments  demonstrate  the  effective¬ 
ness  of  the  on-line  use  of  computer  software  to  create  and  modify 
software  routines. 


n 


4 


\  V  'f'\  . 


a\  report  on  Contract  N0001  4— 79— C— 073‘.\ 
-1t;1  Lviwoen  l\rformance  .Vleasurement 
;.tV!!vorino  Psychology  Group,  Office 
oorv.oact  war.  initiated  on  1  Sept  1979  and 


'  invent  dialed  the  capability  of  programmers 
■  o  specify  nroblem  solutions  by  developing 
;rb:ro!'v;  ana  a.  o  by  writing  computer  programs;  each 
•bicat  too  w  ;  accomplished  at  various  levels  of 
laxity.  1  \-  level  of  difficulty  of  each  problem  was 


*  ‘  ic at  a 

•n '■  ''li  *x it v  . 


1  by  !  \'  number  or 
n.  Vucmne  n-oce 


d  by  the  user  to  develop 


or.  \\’c,v.?v  tng  of  the  use* — inputs  permitted 

to  ee  bovo!i''\';  about  the  algorithms  reauired  to  solve 
-a!ar  ,»v..vev,  *  he  '.’iterative  feedback  of  processing 

1  '  u-  o*\  to  a  precise  definition  of  the  desired  Solu- 


Six  ox,H‘r"bH'rrs  were  conducted,  with  the  same  problems 
u  eu  m  all  ex-*  .  '  -x%  abbm.v  o'  i  >i.‘  mini  ic  inarm  to 

uevelo^  exo?  •  >lc— ■  -el*  '  tons  was  evaluated  as-  a  function  of  the 
\  •.*"  ’o  m\o:  *  eau'-s  "'icnj  ami  experience,  me  complexity  of  the 
o'o to  'x*  me  K'vl'I  of  !'roci“ ‘.sing  provided  oy  the 

'■-’"'a  'e?\  and  m->  ’  eve:  of  reodbaek-aid?  ,  wlvn  aids  were  available. 


r v* o  • -‘.m-^cal  reports  we^e  Published  and  three  nailers 
%'m  d  v  v.a."  'nti*^  '  mo  experiments  and  results  obtained . 
-  *"em  e.-'o  identified  in  t ho  mb  of  technical  reportt- 
t  t  x1  on  '  of  mis  report . 


x?.x‘*' inert  “?  arx1  .?  were  Jtvo oneJ  to  invent loate  the 
v  x If  %  < ,r  ex*  vm  imniL'r:  and  of  ! >o  o  Vveeper ,/acc ou  ma? vm 

w  m  v«  •  ■ ' X? ' i.' r •  -moarammers  to  develop  oxan'p!o-t -elutions 

’  >  r'  \  x»t  *.c 1  \av\  raws- force  problem.  7  he  experimental 
v  1  ■  *e*'  *xv  ■'  •  m '« 'mmeots  were  problem-complexity  and 

ore  nr  :\  ,  me.  the  anuuat  of  n machine  o recession 


!  ,x? ‘L.,*'i*"e,'t  '  and  4  wore  designed  to  investigate  the 

nbiloy  e*  export  'fa/'^^ors  and  non-orogrammers  to  develop 
acmmate  and  coMV'lra’  •  'xample-solutions  using  various;  feedback- 


n 


r 


aids  at  various  levels  of  problem-erm.' Tx: - n«. 
oosipns  were  baseu  on  tne  results  of  ;_.x  ^  •-'■.me'.: 
tne  systematic  cienurut  ion  of  ox,.;:v;f  i  —  ••  . , 

a  corrlnnat  loail-moa^uro,  iau  sour.  u  '  :«  *. ■.  ■  . 

witr  pert^rnvnnce  (explaining  KY/  of  .  .  »  r.  v*. 

Experiment  ;>  \A/a>»  c'iv\'i> ,  ..  \  ",■•/■  .  v 

of  expert  program me r: ;  to  revise  nm  .  ^ 

tne  form  of  example:  *-so  unions  m  v.  .\(... 

me < 'r r ee  t  ortrk’:>  Vu.  ccen  tr.trov.4i ..  ■  . 

v r  l! > . . •  r  i  m en t b  .  im i  d . 

Finally,  ...  \(\'nnkT.:  v-  erf.  u  ...  .  .  - 

to  uevelop  computer  cosiv*  written  m  ;  \  %  n  **'- 
levels  of  dm a  ir  x.t  -  a  demon  ink  ,e-, .  .  .  v  ..  ... 

desipn  of  experiment  1  .  Tne  resTo.  .  :  .  v.  . 

sub-  rou  1. 1  nes  w  r  i  it  on  i  n  FORTRAN  i  V'  w  . 

a  snip  combination,  as  that  combination  w. .  .  *  ^  v\ 

The  performance  measures  i.soa  in 
sisted  of  error-measures  and  sfratCvj um-me^. *v: 
measures  were: 

a.  ,  the  orcoaoility  tr. it  e.  owm  sv.o- 

i  > 

was  correctly  Ciassifieu  as  v.Cva'.\...'’.v 

b*  ,  the  probabiluy  t.nat  a  oorroe. 
was  accepted. 

c.  P  ,  the  probability  that  an  incorrect 
ation  was  rejected. 

In  addition  to  Iho  error — measure..-*  aoove, 
error-measures  were  used.  A  relative  srror-meu: 
defined  as  a  participant’s  orror-'3corc  'F__  ,  »  ,  \ 

an  experimental  problem  minus  his/her  error-  . son 
test  problem.  The  relative  error— measure: *  thus  u 
move  the  effect  of  the  participant's  maa-  canu.  »w;t\ 
result,  were  more  sensitive  to  experiment  ic;  ors  i 
error-measures  alone. 


i 


r 


i 

5 

v 


Two  :  •  r  r  a  t  e  *  iy-  m  e  as  a  r  os  w  e  re 
wun  winch  participant,  a.-ou  specific  mo 
sure  ,  the  comoinational-ncar-^rij,  detx  v 
a  participant  enanget  only  one  .'ompomc, 
cxamf)ic-i.'Olut ton .  Another  :  :raa gy-nr 
detected  tne  ase-pattern^.  of  the  vuruxe. 

><ESU;_f  : ,  v  ;  k  _x.  f  ;.\\  \ 
."a  1  .  ar. 


cate-";*  ■me:  •,  arc  dose  raw,  .v-Iev\  .  .a  .*•  . 
tnrougn  c.  war.  largely  case...  r"  .  mrr  . 

t  'roc os.  .or  Corrnlo\r\-  -m;  '  c r 


i-  irst ,  at*  expected ,  more'  error.  •  tv  .«rm  ....  re  *  •• 
work  on  tr.e  more  complex  g  xvlc —  .  aw  ever ,  v  d  v,-‘. 

procossinoj,  or  gene  rale  at  toon ,  of  me  •  \am  ole-.  *(  >lm  ,  ■.  •*  a  ..  .Tmr. 
to  he  an  important  erroi — reuse  m.g  factor,  i.e.  ,  a  .agrafe  a 
reduction  in  errors  occurroti  wren  uato  from  example—.  .elm  a  m 
were  processed  into  a  standard  form.  and  presenter;  to  me  par:  v.- 
pant . 

Systematic  Strategies  and  Fccdback-Amf  • 

A  second  result,  and  pernags  tne  mo;.;  irapi'-aar.;  ,  was 
that  participants  in  botr.  categories  wr.o  performeoi  wed.  tenar.. 
to  use  a  systematic,  step-by-step  strategy  m  selecting  ex.. mire- 
solutions.  This  result,  together  with  the  first,  notes  ac-ovr, 
suggested  that  feedback-aids  might  he  dosK.r.ea  to  o!va..raaa 
participants  to  use  systematic  strategies,  by  iwco.-.ar,  m.m.r 
example-solutions  and  then  feeding  nock  tne  resultant  data  to 
suggest  possible  additional  inputs. 

Breadth  vs.  Depth  of  Experience 

A  third  result  of  the  first  two  experiments  aorlmu  to 
the  subsequent  experiments  war*  tn.it  me  number  cf  yec.n  •  ajvanov. 
education  (i.e*,  beyond  high  sciioo l)  and  tne  numoer  of  years 


:  •  •  < 


of  professional  experience  were  fo^r...  10  .  <*. 
in  predicting  performance.  As  a  con.  .cc^er.cv  c  ;  \ 
additional  demographic  factor^  were  ^  v^L.  cl  a  f.  r 
in  the  subsequent  experiments,  in  an  ef.Vr:  :o  rev 
predictors  of  performance . 

Low  Frequency  of  Error^-of-Comr' _ - 

The  fourtn  result  applies  to  is-  .o.u  ^ 
the  observation  that  only  a  few  error -« ;  f-com. m  . 
during  the  generation  of  toe  example-  .1  .  . *  n.  .  7 

errors  that  did  occur  were  error^-o; ■-» .»  - ;  n,o. 
result  influenced  the  design  nf  Exp-.-nni  ,  w 
code  was  written  to  solve  toe  . ^arrv  .  .  '  ■**..  ... 

1  ,  so  that  a  comparison  of  erromn,  v.  -aw.L.  .  . 

Expo^tr.i-  :: 


h eedback-Aids 


Three  aids  wore  o  eve  lose...  A.  ,  *7  no  vs  .•  , .  '  -  - 1. . 

selection  logic  (SSL)  implies  oy  t."v  .  or;.,  ox^mo.  T- 

tions.  Aid  #2  induced  the  SS„  one  or  or.  .c  o  o  c.-  ..mo  .  os- 
type  of  the  example-solutions  input  ov  ;  a.  same.  pant, 
ordered  listing  was  mtenaed  to  a. a  toe  participant  oy  .ee’s.o, 
possible  omissions  in  snip-combinations .  Aia  -o  indue  '  .  o*o- 
SSL  and  a  list  of  suggested  ship-comamat. cr.o  to  consicc  r.  Tre 
suggested  ship-combinations  were  loose  logically  'vq^irco  to 
complete  the  combinations  suggested  oy  toe  cxamplc-ooLtinm. 
previously  entered. 

The  effect  of  the  feed oac>  aia.»  ot'  measured  oy  toe 
error-score  (P^),  i*e.  by  the  probability  of  accepting  a  correct 
ship-combination,  was  not  statistically  significant.  ■  iow ever, 
the  effect  of  the  aids  as  measured  by  t'  relative  error-score 
(error-score  on  the  experiment  problem  minus  tre  error-score 
on  the  pretest  problem)  was  found  to  oc  significant  and  imnortani 
Apparently,  the  feedback-aids  did  help  at  least  a  portion  of  toe 
participant  population  —  the  Icss-tnan-superior  performers. 
Those  who  would  perform  well  without  lr,e  aids  were  not  helped 
by  the  aids.  Also,  it  was  apparent  tnat  variations  in  performance 
due  to  the  participants1  innate  abilities  were  greater  than  varia¬ 
tions  in  performance  due  to  cne  feedback-aids .  This  factor. 


4 


plus  the  observation  tnot 
use  tne  aids,  may  accoun*.  for  ; 
aids  on  error-score  ana  t  c >  s ; 
score . 


rror..>  r.;.y  not  - 
•ificar.t  effect  of 
ri  i  I'd  on  retGtiV' 


F eedoac k- A  * a  -  3 
creator  degree  t 'an  A,  . 

I og i c a  1  e x am, p  1  e - u ol •* /.a  .n . 
am  pie- solutions  input  pro- vs 
eluded  an  ordered  Is  t  of  cx 
Wit1*  Aia  -2,  the  participant 
i  npu'-  an  * .  cent  i  fy  n\  m  i 
too  part  icipnr.;  wo. .  .  r.v.<-r  ... 
example-:  elution-.  pr. -v;  w  . 
reasonable  to  ex  pec  '  A. 
tnan  teat  supported  sy  a\s. 


:  :C  .1 

A.  .  : 


o  C  :  et  perrorr. 

■  .wc.  revse,.  ;• 
.  n  (. 1  *  ~  c  *  r>  i  1 1 1 


:ou 


IVt  • 


Aid  -o 

without  indicating  :  c  sac..  f--*r  :c.  e 
vo  r  I  ns  t  an  c  c ,  A  i  d  -  3  w  a-  u  i:v.  v.  4  r  1  y  ; 
a  few  example- solutions  \*u.  i  cr 
were  possible  next-io.  c  acc~:c  v, 
Also,  when  tne  participant  an 

tamed  an  error,  Aiu  would  recom1 
complc  wed  tne  error— induced  pattern; 
thus  be  either  helpful  or  harmful,  ce. 
When  recommended  solution^  arc 
this  type  of  feedback-aia  is  potential! 
display  of  patterns  built  upon 
of  the  error’s  detection  -  by 
provide  recommended  solutions  may  help 
errors . 


tot 


error  mcreo 
aoyir 


r.;  -Oi  u  1 1  Or.s  t  r .  s*:  t 

w*  V  tA  iCo  CiS  /~V  .  V  , 

name  on  now  tncy 
w;  trout  careftil  ey 
harmful.  If,  x-wc 
es  the  like* 
on 


vo? 


.u'Cv, 
VwU~  n  , 

■ ,  t  n  t. 


ts  impact  - 

'■c  user  to  acta 


Rreadth  vs.  Depth  of  r.xponeneo 

The  lack  of  a  strong  predictive  relationship  between 
years-of-higher-education  or  ycars-of-expericnce  and  performan 
may  come  as  a  surprise  to  educators  ana  directors  of  per.  .anno  1 
departments.  This  result  was  found  m  all  of  the  experiments,  so 
that  very  strong  evidence  is  available  to  support  the  assertion  tha 
years-of-education  and  relevant  work-experience  are  not  goou 
dictors  of  problem-solving  performance. 


Additional  rc  It-  •  ...  .  t  ;  «■  -  ’V^rr.  ur  r 

r.mng  languages  (used  on  1  or  more  pre.'mvr;  f' T*  /.r.a  ’Vw 
operating  systems  ;,n*  .vror  pr.  -.;.  of  c ,t 

of  computer  user.  ./on  c. nan  -  owm  . 

Combinational  St  rat-*  :v 


C o m. o l r. a *  r  w  •  /  w . ;  * f 
predictor  of  performer.  -  * .  .  **'-  .fa  ,  - 

Experiment-  1  ami  2  ,  wo-  ■*-  .  c  a.  ^  x.m  m 

variance  war.  explain.  , .  ;  \  cm. -'-.an  a  . ;  re.  I-., 
merits  1  and  2  ana  Sm  '  m  ;  m 

the  comoina;  loral-: c  r  v  .  •  ;  .  ■■  -■  t  ,m  *<  -’.• 
to- moment  measure;  u  •  om  rn*  tow.;- . 
mom  uni- to- moment  to;  ...n  p.  *rm  c  1  ..  • 
lateci  to  performance  an,.;  t  gr*. 

sensitivity . 


F  e  c  db ac  k— A  i  ds 


In  Experimor*  f,  p,mt m g  mm.  wv* re  asxea  ~r  m  c: 
example-solutions  tool  me  U.  den  vnno*..  r.  m  p  m  of  n^> 
combinations.  This  exporter  t  do;*  tgn  oerm.ittoci  calculn  nor 
the  probability  of  maintaining  an  ;m;  .slop-correct  exam-'.  - 
and  the  prooewnuy  of  detecting  ana  correcting  an  initial c,  -m. 
example-solution . 


Analyses  of  orana-mcan  prooant  1  itie^  of  sjccc  m  v-  f. 
that  the  probability  of  maintaining  an  mmally-corrcct  odeum  aw 
.93.  However,  there  wj;-  a  probability  of  .05  for  detectin',  am. 
correcting  ac.  erroneous  example- ..olut ton.  Obviously,  ~e  w.w 
a  performance  decrement  in  ;  o  detection  and  correction  e:‘ 
erroneous  example-solutions .  Useful  feedback  -aids  rru-rr, 
be  directed  toward  the  detection  ana  correction  of  existm  ,  errors 

Feedback-Aids  #2  and  #3,  which  were  the  same  a  -  tr.m  t 
used  in  Experiments  3  and  4,  were  not,  however,  a  benefit  to 
participants  engaged  in  the  revision  of  example-solution^ .  fhwnc 
an  analysis  using  relative  measures,  wmen  tended  to  remove  me 
effect  of  participant  skill,  did  not  result  m  statistically  m,  mifican 
results.  The  conclusion,  then,  was  that  tne  strategies  useti 
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.fully  :o  revi 


t  ha  t  t  aa  a  i  cf  >  d - v •  ■  i r.i\,  *.c.  '  e  I 
woro  no:  nelpful  m  mv.  m  ■ 


;,n  ;  ,<  »km  ;on. 


•m.-.  r.k  .* 


;oia: iv:ry  of  terror 


i  n  detect  i  on  of  *  ■  nr  or  ■  .  A 
solutions  wa- ■  aecrr-n.  •  n 
do  t  ec  1 1  ng  a  no  cor  roc :  i  r 
fewer  the  nu m  c  e r  or  •; 1  r r c 
detect:  ng  a  nivo  n  err.--. 


i  me  result  •  s  m  v  * 

probability  (cased  on  :  s-  f  reem, oncy  «*f  i 
of  detecting  and  corre:-'  ;rv;  . error  . ,  wr  i c 
of  judging  trat  any  solvuer.  moor  roc 

of  a  solution  as  dotermin. -a  oy  a  ^  or  r 


m  net  ion  o: 


1  .  Tne  actual  corm 


o:  '  ,a*  rmmior 


2*  The  peromveo  frc:> 
f ou nd  roc  e nt  1  y . 


•  r r o - e a  net 1 or 


This  hypothesis,  consistent  worn  nrv.acf.o-.  of  moral  detection 
theory  (which  says  tnat  r  m  pronaoility  of  ;,r  oven;  m  a  function 
of  a  base-line  probability  ;n  aaamnn  to  m-  m ific  moasurerr.cn ts 
on  the  signal  itself),  predict :■>  a  decree,  m  ,  croc, anility  of  detecting 
and  correcting  errors  witn  decreasing  or --nr- rates .  This  moan. • 
that  the  probability  of  detection  : -s >  la  ;  dw  terrors  may  bo  to 
small  that  it  is  not  likely  mat  mey  will  o«  ■  found.  Eut  it  also 
suggests  a  possible  solution:  uoceur.a  e  rrors  (which  car  later 
be  removed  if  not  detected)  to  increase  me  oase-l  me  erro-^  rate 
and,  thus,  to  increase  the  prooaoihty  of  detecting  unknown  errors. 

Experiment  e 


Two  types  of  errors  wore  analysed.  Cnc  type,  terms 
an  "error-of-omission",  referred  to  an  error  that  resulted  m  a 
failure  to  accept  a  correct  entity  (e.g.shio  combination) .  W-mr. 
specifying  a  problem  solution  with  example-solution^,  nr.  error- 
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of-ormssinn  could  :  >•  *  e;r.  .  :1\ 
of  a  suitable  entity  (  i . 

'  cm  w  .  i;  it  mvi  i  wa: .  at"'.  Mrrr.  ■( — i 
t  l  ( ~<fV  i  Were  vi  ‘,c*i  to  :  ;  m  .  .  '  ■ ,  .  .  .  ' 

nis.'ion  corr(v>poauv.  ••  an 
proroaaor  whies  wa  ‘ 
example.  An  ^rror-i  ’  -  •  nn 

no  *  -opt  mo  inrorr-v:  •  •:  . 

t.>ror’  --  of-  On ^  \ :  ■  a  tar 

T  fvro  wa.  a '  . 

complexity  on  error  •  4 

. .pe.  ifying  problem  ■<  •!>,'  .rr  , 

POrCl  i\ANi  IV  ; 

Lrrtr  -I'f-Cnrr.n  »• . 

Wno n  :e.oe  r  ;  n  »  - 
aid  ,  ‘  i  a  •  rate  of  i*rr»  -r  -  o; < 

;  H'  •  •«  •  I  ■  ■  .'nn'n'n'xr.y-!.  ■.  t 

i  l  i Vo'  ate  •  *  :  \  r  ,  ■  ;iv*  •"  .  ...  * 

nr.  in  Exponmon'  •  >,  4  a  .  r.  . 
eliminated,  a:--  ev;d-  me 
f  o  r  m a nc o  do : r ac i a '  1 01  • :  t  ;  r  • 

The  mo.;  i:v;-,or  ,  .m  a-  _  '  .  .  .  r^nr.  -c;'-.' o  " 

mission  was  that  apov  ;f .r  ny  i •  >,  in-  ...  -  <  ..ran  wa  >  ^ 

to  specification  by  pro -mam  v  o.  .<  •.  Ar„*a;  i  .  ;  *  a  moan  rnr.  • 
from  Expcrimot.tr  1,  ana  a-  v;  -  .<  .  •'<  r  ,  rvia’rvo  :  ~v  .. 

example-solutions  sue- an:  ia'dv  ■  .  .-rror*  —of— com  mi 

compared  to  using  FC)<  ]  t<A\  iV  an*.  ;rum  ,  Tr.e  dd  r.v  o  : 

crrors-of-commiss ior.  wi:  '  .  x. .elution  .  com.paroa.  favor 
with  18%  for  program  roue. 

Three  hypotoeoeo  concerning  fr.o  superior  porformn 
of  the  example-solution  met  nod  seem  plane tele: 

1  •  It  was  working  witn  examples  and  dealing  with 
each  individual  combination  of  item:,  ono-nt— a 
time  that  result  oil  in  a  low  rate  of  errors— of— 
commission . 

*For  a  discussion  of  I  laletead’s  L_  Mclruy  See  Connelly,  Com(Mi 
&  vJohnson,  Technical  Report  81-3M  ,  1t>M  . 


: .  it  wa  •  :  •  .  ..  • 

U  v'otv.p.arr  ;a 


a ;  (  >i  n  *  vv.  i 


■  <>lm  H  ' 


Sifters:':  , 


u ..or  .  ( >  v . ».  ■  w  a'  sr. 

tnat  re  . .  m  . 
Con>f. p  .u  rt  ly  ,  .  ’  pr 
won '  *  r. .  arr.'k  •* .  ; 
fed  :,a.'iv  :  o  ; ..  e 
error  *-c.f-com  m 


(  r*  v.'. 


-(  :  — ...  i  ,r  ~  rr ; 


. o  (to  .  •'  «r  •  *.r 

a  low  rati'  oi 


TacoO  nyp.t  are  r.i  •  :  a  a  ^  a.  at  ive  nypomose.  —  a., 

could  be  true.  We  navi-  tor,  ;  rv.ei  *n*  e  t  o  t  a  ter.:  nyrraam. 
is  true.  If  me  see  one  m  trv.c  .  m  r.<  :  a  ..  ,r. , ,  ;  .roaram-oosign 
ana  coding  methods  auapm -a  a  ■  a  m.rv  ;a <m mr..r.  tor- 

f 'ependent  structure.  Ana  ; mam/,  ;f  aa-  o.;r«;  nypot  a; -ms  v^ro 
found  to  be  true,  pro-r.on'.t  a  ia:  a  >r.  ..a  >.  .  •«  •  :  *  •  *  ,:~vv 

the  user’:',  program  roui*  into  an*  >t  a  r  iYrr  vw  ;le  rr.air ;  a  mine; 
the  same  program  logic')  f»  *r  iVvoo.'iok  t o  ~e  w.  or. 

C.O\C;a  o,iC\b 

I  .  Fecdback-am  ,  u>  support  u^e  of  oxampU-  au.o.or,  , 
should  include  me  logic  implied  oy  the  examp  m- 
solutions  .as  well  a--»  new  OAarr.pim.oluiior..  •  mm 
complete  mo  logic  pattern. .  ..uggosteu  ny  me  oxi  ‘.mu 
set  of  example— solutions . 

2.  FecdbacK-auis  consisting  of  an  ordorow  iL»:in.,  i  . 

all  present  solutions  also  supported  mgr.  pi -r:  -  rn  *no  i 
but,  the  predictive  aid  type  referred  to  above  m 

preferred. 


cm  t  a  firm  gyro:  e,e.  m 
.  .r.,,  pro;:rar.“-oosign 
: -e .  »n  c ( .m i r, , a :  i or  — 
-an.;  synop a:.- u>  'Aero 


^  i 


Feedback-aids  as..c  .  r'.  .r 
of  bom  programmer:,  . ;  *  < v 
not  perform  well  wu  io. 4 

Performance  measure:  •  >;•  . 

performance  improvemra1  v. 
b-'  relative  measure  we... 
between  performance  or,  a  cc 
oxperimen, al  ta  - k . 


The  lack  of  a  strong  r* -la:  ion  -  a  1  p  --ween  -of 

higher-education"  ,  ’’year • of-exnsr  i*  v:v "1 •"  a-  ;  ;  ir- 
formance,  coupled  with  t.ne  stron-;  rpl(r  nr, 
between  "number  of  computer  Innaung*  •  h*  i-es  wr  n r 
"number  of  operating  ayotemi"  usoo,  v  >  o 

education  and  experience  should  r.o:  no  ,-  v,,  . 
they  have  been  in  the  past  for  hiring,  ;:rrr>  i“"  , 
determining  salary  level,  and  a  :a  .k  . 

Instead,  the  number  of  corapvj*  or  language  v.'i '• 

and  the  number  of  opera:  inn  .y.  .tem:  •  u,  \vn;r 

arc  bed  or  performance  pr-  -.ador  »  •  .h.  ; • 

used  until  I i ao  underlying  fan; or  .  inck^ii-  .  : 

are  ch: '.cnvi'roci. 

Apparently,  the  depth  of  an  inaividunl's  exner  a  iivi  ■ 
is  not  as  important  to  performance  a.  »  •  t  e  ure'.w:, 
of  his  experience. 

A  possible  common  underlying  exper  ter.ee- re ,  a «  o 
factor  is  the  ability  to  view  problem  from  ,\hr:v(- 
tive  viewpoints ,  or  the  ability  to  develor*  .f  ■•■r,;v  ;v' 
approaches  to  problem s  -  an  ability  that  mi-p,:. 
enhanced  with  feedback— aids  • 

The  performance— prediction  capability  of  .  ‘rate  gv- 
mcasures,  developed  as  mommi-io-momcn;  mea¬ 
sures,  not  only  clearly  demon:.tra:a,  that  sy  eu-ma: 
strategies,  wore  used  by  succo. •  .:ul  part  icia, ,ne. 
(wnich  led  to  the  design  of  the  feedback  au:.. d,  nut 
also  convincingly  demonstrate:'  tr.at  momont-io- 
moment  measures  provide  me  sensuivay  to  ex¬ 
plain  considerable  performance  variance  (approxi- 
mately  60%  in  Experiments  1  tnru  4). 


r 


9 .  When  m, ih  ;  l fy ; r.  .  .  m al  ly- 
thc  [''roiwo  lilt y  ( . 1  Th  c »f  r. 
eorrer;  « 'xarr.i'lc-  ,r,lm  ;o: 
as  tee  ;  ■.  m:  il i  fy  of  s» -v- 

exarr.pl  • — .  .oiu:;«‘n, 

Jottv;  m  ;  v  u\.;  ;  or***.  -c'  <r 
:  .olu:  lor.  was  sw  v.  k 
shout-  .  se  deveas.ea  , 
i’nrrer;  ion  of  ^xi  i:*m  •  r 

10.  The  pnWu'ili'y  of  deb  ;  ;n  .  ;(r,;  .  c  ^  s-cer  ,  nr, 
ornnTHi:'.  example— s<  -he  : on  •  j<  v  -s  . .  <  ■-  (  . » 
initial  n;im;xr  of  irrona-  exam.m —  rh‘  ;nn  wa  • 
Jccrra.  >cu.  Tine  fewer  :  :s  •  ;  o:  al  m.m.  of  .  r^n*'* 
the  le  •  ■  wa:-  i  no  I  iKel  ir.oo«..  c  <:  v;e:  •  v: :  ;  r a  ever,  •  ^ror. 
Thiii  ,.Uv]qosi:.>  teat  a  metro*.,  for  ;r,:r.a  ,iiv,  :  s* 
probaoility  of  detecting  error;.  is  to  .-oea  error-., 
unknown  to  the  individual  reviewirn  the  example- 
solutions  (or  cor:puk'r  nro-irar';  out  oircmv 
rocor«icvi,  to  incn’a:  o  tr.e  .  e.  >e— line  r  is  •  e:  < 
detect  ion  ana  therefore  ;o  increase  :  so  .  roea:  dry 
of  decoding  an  unknown  error, 

1  1  .  Software  err  or- c  otcaones  arc  typicaiiy  aei  ineu 
only  to  facilitate  v.iata  co’.iecaor,  ana  n v  -ram.',. 
However,  analysis  of  .  of;wam •  errors  :  aowca 
that  when  an  error  co;erjc*ry  wa.  docomponed  into 
sub— categories  (he  md«  *j mt  variables  in  t. '.e 
prediction  equations  cr,<\nc;<\. .  1;  is  concluded, 

tnerefore,  that  •  svdware  errm — caioporich  should 
be  selected  with  reparci  to  predictability  as  well  as 
to  data  collectaoility .  Two  sub-catcgorics  should 
be  combined  only  when  tneir  prediction  equations 
are  homologous,  i.c.  have  the  same  independent 
variables . 

12,  The  superior  performance  (fewer  errnrs-of-com- 
mission)  achieved  when  u  .mg  example-solution:, 
and  inductive  processing  to  specify  problem  solu¬ 
tions  over  the  performance  achieved  when  using 
FORTRAN  IV  code  may  \ provide  a  basis  for 
determining  the  underlying  mechanism  for  that 
success  and  a  moans  for  incorporating  that 
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mechanism  into  pro.  a  •  ■  ;r.  ;r  .  c-s ; .m  ;-,v.  e: . . 

Apparently,  :-uporior  p.  -re  vs  -  .v  .  r... 

either  because  ea,  '  ,  t  -•  .  ;r  mn  o;  ;  r. ■  m;s.:  var .  '.>!«.  ■: , 

was  treated  individual  !y  w./j  .r  .  •  c  ex‘*rr:~:le- 

:  ,olui  i*  >r,.  •  w^rc1  ;ranMi'r  „'',  :  r *  t  *  :,ro:  s-r  lr  .a  rr  rr,  — 
tho  •  .-up  .'.'i'Vt  inr,  1 1  *  ,\c  d  *  •  .  Y^-'-a-r  ..  a 

si']nn\\in!  :\«c:or,  i  ."iiT,  •  ur.:-  .  m  ,  -.or. 

shoulu  re  adapm-;  t  o  pro  ;r  \r  ;•  •  -•  :rm  .  ..  or .  m  ’  ;- 

aids .  If  '  n»  ‘  latter  i :. »  a  ■■  \ gr.* ;  •. \r:  v  .  ;or  1  '■  >.  ■  ;  ‘.nr. 

ana  ooaina  -aid  .  -souls  r.  .» -v«.  'ion.  ■ .. .  *  r  '.a.  »  >rs~ 

the  logic  nrnvi  so*  i  :>y  me  u  .  m  ir*  o  amvoer  ;V^r  wmes 
i  .  t i *n  *n  t  e< «  .  ?s  i o  *  ■ '.» *  u'-er  t  ("»r  air.*  r-'vs  w.  -.e 
ir.in'.fnrivafinii  raa  i  a;  *  pn'M-n:  ;  •“'e  f^rrarr.rr'  .  *  d  n 

Incur  . 

RECOMMiiNi)A'i  IWY  i  OY  r  UiRTi  ii_Y  YlsSE£ ARGi  \ 

1  .  Tno  strategy-measures  uca;  to  analyse  FORTRAN  IV 
wore  not  moment-to-momer.t  measures.  Instead, 
they  w crc  a  classifies; ion  of  typos  of  possible 
strategies.  Tnc  predictive  power  of  the  measures 
was  only  moderate  compared  to  those  used  to 
evaluate  performance  in  developing  example-solu¬ 
tions.  It  is  suggested  mat  moment-to-moment 
strategy-measures  be  developed  for  both  program- 
design  and  program-code  tasks. 

2.  Feedback-aids  designed  lo  support  development  of 
original  example-solutions  were  not  found  applicable 
to  the  revision  of  erroneous  example-solutions . 

Since  use  of  example-solutions  is  a  viable  way  to 
specify  problem  solutions  in  itself,  and  reveals 
ways  of  improving  program  design  and  cods, 
successful  revis ion-strategies  should  be  identified, 
and  revision-ai  js  should  be  derived  from  those 
strategies . 

3.  The  "number  of  programming  languages"  knowr. 
and  the  "number  of  operating  systems"  used  have 
bean  shown  to  be  good  predictors  of  performance 
both  in  developing  example-solutions  and  in  writing 
program  code.  It  is  also  suggested  that  me  ability 
to  develop  alternative  approaches  may  no  a  common 
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factor  which  may  be  erc.anc:*  by  learning 
languages  and  operating  .yes-m  . ,  ar.  i  wnics  n~.r, y 
bo  a  key  performance-fact  on .  ““  rat  *  ^nd*  *riy;rv; 

factor  or  factor:  ,  result  in-  :  n  ...:  vrior  p.  -rfoernen  ■*.  * 
should  do  determined  to  a.  ..a:,;  m  per  ..el  .  .elects on 
and  design  of  improv  a  aide. 

It  is  suggested  mat  error-doles  ‘  s  -r  .  ns -a:  fesy  -".a'/ 
bo  a  function  of  a  ba^e-lire  error-  y  r.v  , 

and  therefore  mat  ..•eeUir  .  ..  m-v-*  or*%rnvvr.  ys  *n 
individual  cnecking  r.r.e  mas  r :ai  nay  a  -  *  v  * 

probability  of  defecting  an  %  .n.  .o«  •<  j<  •* ;  -  rrr.r.  v. 

Con jec  tore  ',i',nui-J  e*  •  .a»  ,•  ••  s  •«  ;  <<  <  y;  .  ns 

t  os  l  to  t  Jet  ermiiv  *  if  -eed::.  ;  ...  j  -a  -rd  ne  an  <  •  > 

and,  if  :<o,  what  frequency  and  ;vpe  of  .ee. ,  enmn- 
should  be  used. 

The  basis  for  superior  nnVmarce  w.t  '  example- 
solutions  needs  to  he  resolved.  ‘7  ',e  concnv.  of 
writing  program  code  for  ■  -arc  comm  nation  of 
factors  (or  the  use  of  an  to  automatically 
analyze  logic  to  help  develop  accurate  eommnataon- 
independent  logic)  and  mo  concept  of  coco  trans¬ 
formation  into  a  different  logical  form  for  feedback 
to  the  user  for  approval  need  to  he  con;  raster:  in 
an  experimental  environment .  Tnerc  is  a  potential 
here  for  substantially  increasing  me  correctness  of 
computer  programs  if  an  aid  can  oe  developed  to 
permit  transfer  of  the  superior,  almost  error- 
free  performance  witn  example-solutions  to  code- 
writing  performance. 

Independent  of  the  methods  for  improving  perfor¬ 
mance  in  writing  program  code  suggested  in 
Recommendation  f>,  a  now  aid  design  should  be 
considered  that  combine:  ,  general  statements 

written  in  program  code'  with  redundant  example 
solutions  —  i.c.  with  example— solut  tons  that  are 
not  part  of  a  program  test,  but,  instead,  that  are 
inductively  transformed  into  an  alternative  code. 

A  pre-compiler  nu.  A'ould  produce  actual  code  from 
both  sources.  Potential  performance-improvement 
could  provide  high-quality,  almost  error-free  cooe . 


aoknow!  .r^n  r'Mi  at:  , 


The  author  is  qratoful  to  Robert  P.  Con 
tributed  siqnificnntly  to  iho  development  of  tr.r  itv.- 
and  who  wrote  the  system  and  analysis  program:,  a 
Mis  interest  and  enthusiasm  for  the  work  eon:mov.:< 
success  of  the  project  .  The  author  1  .  ah  o  qra:efu 
Johnson  who,  while  with  us.  for  only  a  .nor:  time,  , 
positively  by  conducting  Pxporimene  and  -k  an*, 
data  analysis  for  those  oxporuv.enis .  Pur:  her,  p-e 
qrateful  to  Mike  Ohshausen  for  dediea;  ton  :o  nr-  :  a. 
throughout  all  revisions.  Finale/,  eu  au*  or  is  ;r 
i  * .  John  vJ,  OMlare  and  i  ^r.  /VMr;  m  i  iTi>:i  ft  »r  :  ■  ■ 

interest  in  the  effort. 
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